An example for a multi-view video codec supporting residual prediction is 3D-HEVC. In the current design of 3D-HEVC [5], the coding unit (CU) decoding process for residual prediction ([1], [2]) refers to three reference pictures. The pictures can be identified by a reference picture order count (POC), a reference view order index (VOI), the current POC, and the current VOI.
The reference POC is the POC of a picture which is a) included in the reference picture list RefPicList of the current picture; and b) included in the current layer. The reference POC is derived from syntax elements of the VPS and the slice header of the current picture and is constant for all CUs of the current picture ([3], [4]).
The reference VOI is the VOI of a picture which is a) included in the reference picture list RefPicList of the current picture; and b) included in the current access unit (AU). It is derived from syntax elements of the VPS, the slice header, and the current CU and can thus vary for CUs of the current picture.
The three reference pictures referred by a CU are included in:                1. the current view and an AU with the reference POC (denoted as picture A)        2. the view with the reference VOI and the current AU (denoted as picture V)        3. the view with the reference VOI and the AU with the reference POC (denoted as picture VA)        
When a reference POC and a reference VOI can be derived and all three reference pictures are available, residual prediction is enabled for the current CU.
The availability of picture V and picture A is implicitly guaranteed since:                The reference POC and the reference VOI are selected among the POCs and VOIs, respectively, of pictures included in the reference picture list (RefPicList) of the current picture.        RefPicList includes a subset of pictures included in the union of picture sets RefPicSetLtCurr, RefPicSetStCurrBefore, RefPicSetStCurrAfter, RefPicSetInterLayer1, and RefPicSetInterLayer0, which are derived from the slice header of the current picture and the VPS.        Pictures included in RefPicSetLtCurr, RefPicSetStCurrBefore, RefPicSetStCurrAfter, RefPicSetInterLayer1, and RefPicSetInterLayer0 have (by a bitstream constraint) to be in the decoded picture buffer (DPB), when decoding the current picture.        
Picture VA is available, when the decoded picture buffer (DPB) includes a picture which is marked as “used for reference” and has the reference POC and the reference VOI ([3]). Thus, whether the decoding process for residual prediction is invoked for a current CU depends on the state of the DPB.
Note that, although not explicitly tested in the decoding process, the picture VA is available when both of the following conditions are fulfilled:                1) The bitstream contains a picture with reference POC and the reference VOI.        2) The syntax elements of the picture V indicate that the picture VA might be used by as reference by one of the following:        a) Picture V. This is indicated, when picture VA is included in one of the sets RefPicSetLtCurr, RefPicSetStCurrBefore, or RefPicSetStCurrAfter derived from the UPS and the slice header of picture V.        b) Pictures following the V picture in the view with the reference VOI. This is indicated when picture VA, is included in RefPicSetLtFoll or RefPicSetStFoll derived from the VPS and the slice header of picture V.        
Note that, without loss of generality the description above and in the following assumes, that the current CU and the current slice only performs P prediction. For B prediction the process                might be performed for both lists of the CU.        might only disable the use of a picture VA when not available, instead of disabling residual prediction entirely for the CU.        
An Example 1 is shown in FIG. 1, which depicts layers of a 3D-HEVC multilayer bitstream and picture dependencies of CUs of a current picture. The current picture is picture 2 of view 3. The parameters sets VPS, SPS; pictures 0, 1 and 2 of views 0, 1, and 2; pictures 0 and 1 of view 3; and the slice header of the current picture are decoded before decoding any coding units (CUs) of the current picture.
Moreover, before decoding the first CU of the current picture the reference picture list refPicList is derived from syntax elements of the VPS and the slice header of the current picture. The list refPicList includes pictures 0 and 1 of view 3 and pictures 2 of views 0 and 2. The reference POC is selected among POCs of pictures included refPicList and view 3, such that it is equal to 1.
When CU 0 is decoded the reference VOI is derived from syntax elements of the VPS, the slice header of the current picture and CU 0, such that it is equal to 2. Hence, pictures V and A are picture 2 in view 2 and picture 1 in view 3, respectively. To determine whether residual prediction is enabled for the CU 0, it is furthermore tested if the picture 1 of view 2 (picture VA) is included in the DPB and “marked as used for reference”. Since picture 1 of view 2 is present in the bitstream and included in RefPicSetStFoll of picture 2 of view 2 this is the case and residual prediction is enabled for CU 0.
When CU 1 is decoded the reference VOI is derived from syntax elements of the VPS, the slice header of the current picture and CU 1, such that it is equal to 0. Hence, pictures V and A are picture 2 in view 0 and picture 1 in view 3, respectively. To determine whether residual prediction is enabled for the CU 1, it is furthermore tested if picture 1 of view 0 (picture VA) is included in the DPB and “marked as used for reference”. Since picture 1 of view 0 is present in the bitstream and included in RefPicSetLtCurr of picture 2 of view 2 this is the case and residual prediction is enabled for CU 1.
A problem of the current residual prediction decoding process is the dependency on the presence of a picture with the reference POC and VOI (picture VA) in the DPB. Such a dependency on the state of the DPB is in general avoided in HEVC specification by design choice. A drawback of the design is that a loss of picture VA is not necessarily detected when decoding the current picture.
Thus, when decoding a current CU and the picture VA is not present in the DPB, it is not clear whether it is lost or intentionally not present.
See Example 2 in FIG. 2. FIG. 2 depicts the same setup as in example 1, with the difference that picture 1 in view 2 has been lost. Thus, condition 1 (as provided above) is not fulfilled and picture VA for CU 0 is not in the DPB, such that residual prediction is disabled for CU 0. However, the erroneous decoding cannot be detected when decoding the current picture, since picture 1 in view 2 is not mandatory to be present. Note that, the presence of picture 1 of view 2 in RefPicSetStFoll or RefPicSetLtFoll of picture 2 in view 2 does not demand its presence in the bitstream or in the DPB. Reason for this design choice in HEVC is e.g., in a scenario without residual prediction at all, to be able to discard picture 1 of view 2 without changing picture 2 of view 2.
Accordingly, there is a need for a multi-view video codec with an improved support of residual prediction involving a picture neither coinciding in timestamp nor in view with a currently coded/decoded picture, such as improved with respect to robustness against transmission lost and/or respect to processing costs.
The object of the present invention is to provide such multi-view video codec with an improved support of residual prediction involving a picture neither coinciding in timestamp nor in view with a currently coded/decoded picture, such as improved with respect to robustness against transmission lost and/or respect to processing costs.